add WasbDagBundle to load Dags from Azure Blob Storage by Nishieee · Pull Request #67016 · apache/airflow

Nishieee · 2026-05-15T23:14:02Z

Summary

Adds a Dag bundle for Azure Blob Storage so Dags can be loaded from a container (with optional prefix), similar to S3DagBundle and GCSDagBundle. Introduces WasbDagBundle, extends WasbHook with container checks and sync_to_local_dir, registers the bundle in provider metadata, documents it in dag-bundles.rst, and adds unit tests.

Manual verification: Tested with Breeze against a real Azure storage account: wasb connection, dag_processor.dag_bundle_config_list pointing at WasbDagBundle, Dag parsed from blob and visible in the UI. PR includes screenshots (Azure container + Airflow Dags list).

closes: #66987

Was generative AI tooling used to co-author this PR?

Yes — Cursor (agent-assisted editing)

Generated-by: Cursor following the guidelines

jroachgolf84 · 2026-05-16T12:28:45Z

I added a note in the PR about enhancing the docs. I think a couple of things that I'd like to know if I were a user setting this up from scratch.

What authentication method do I use between Airflow and Microsoft?
What permissions does my managed identity need?
Are there any custom storage bucket/container configuration I need to keep in mind?
What about networking?
Can I use the same type of Connection that I'd use in a DAG (I know the answer is "yes", but it might be worth calling out)?

jroachgolf84 · 2026-05-16T12:30:32Z

These changes offer a nice starting point: https://github.com/apache/airflow/pull/66993/changes

Nishieee · 2026-05-16T15:22:19Z

These changes offer a nice starting point: https://github.com/apache/airflow/pull/66993/changes

Thanks for the review. I'll add provider-level bundle docs for WASB following the pattern in #66993 (providers/microsoft/azure/docs/bundles/index.rst, Guides entry, cross-link to core Dag bundles), and cover auth, managed-identity permissions, container/prefix setup, networking, and reusing the same wasb Connection as in Dags. I'll push an update shortly.

yuseok89

Left a couple of small comments inline.
Please take a look when you have a moment.

potiuk · 2026-05-18T10:27:53Z

@Nishieee — There are 4 unresolved review thread(s) on this PR from @dominikhei, @jroachgolf84, @yuseok89. Could you either push a fix or reply in each thread explaining why the feedback doesn't apply? Once you believe the feedback is addressed, mark the thread as resolved so the reviewer isn't re-pinged needlessly. Thanks!

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

potiuk · 2026-05-26T01:08:36Z

@Nishieee — There are 2 unresolved review thread(s) on this PR, and you have engaged with each one (post-review commits and/or in-thread replies). Could you confirm whether you believe the feedback is fully addressed and the PR is ready for maintainer review confirmation?

If yes, reply here (a short "yes / ready" is fine) and an Apache Airflow maintainer will pick the PR up from the review queue on the next sweep.

If you are still working on a thread, please reply with what is outstanding so the threads stay unresolved on purpose.

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

Drafted-by: Claude Code (Opus 4.7); reviewed by @potiuk before posting

Nishieee · 2026-05-26T01:15:34Z

@Nishieee — There are 2 unresolved review thread(s) on this PR, and you have engaged with each one (post-review commits and/or in-thread replies). Could you confirm whether you believe the feedback is fully addressed and the PR is ready for maintainer review confirmation?

If yes, reply here (a short "yes / ready" is fine) and an Apache Airflow maintainer will pick the PR up from the review queue on the next sweep.

If you are still working on a thread, please reply with what is outstanding so the threads stay unresolved on purpose.

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

Drafted-by: Claude Code (Opus 4.7); reviewed by @potiuk before posting

yes

jroachgolf84

Nice work, all the changes LGTM. I'm not a maintainer, so my review doesn't count towards your required approvals. But nice job!

jroachgolf84 · 2026-06-09T23:26:30Z

@eladkal - can you take a look at this one? I know that you were involved on the initial issue.

…flowException

…e/bundles/wasb.py Co-authored-by: Yuseok Jo <yuseok89@gmail.com>

… tests

eladkal

LGTM

eladkal · 2026-06-25T07:32:15Z

Probably needs 2nd eye from someone more familiar witht he bundle area

dabla · 2026-07-02T07:16:17Z

+        self.prefix = prefix
+        self.wasb_dags_dir: Path = self.base_dir
+
+        log = structlog.get_logger(__name__)


I would try extending WasbDagBundle with LogginMixin instead of instantiating log yourself. If in the future we would for example say we use some other kind of logger, the WasbDagBundle would then be also updated without any intervention needed.

Hmm I see others do it like that as well, maybe something to refactor in another PR? Maybe BaseDagBundle should extend the LogginMixin.

dabla · 2026-07-02T07:18:14Z

+        super().initialize()
+
+    @property
+    def wasb_hook(self):


This could simply become a cached_property I think, no need to keep self._wasb_hook.

dabla · 2026-07-02T07:20:36Z

+        account_url = self.wasb_hook.blob_service_client.url
+        url = f"{account_url.rstrip('/')}/{self.container_name}"
+        if self.prefix:
+            url += f"/{self.prefix}"


return f"{url}/{self.prefix}"

dabla · 2026-07-02T07:22:23Z

+        self.check_for_variable_type("container", container, ContainerClient)
+        container = cast("ContainerClient", container)
+
+        for blob in container.list_blobs(name_starts_with=prefix or None):


Couldn't this be avoided if prefix would be None by default instead of empty string?

boring-cyborg Bot added area:providers kind:documentation provider:microsoft-azure Azure-related issues labels May 15, 2026

jroachgolf84 reviewed May 16, 2026

View reviewed changes

Comment thread airflow-core/docs/administration-and-deployment/dag-bundles.rst

yuseok89 mentioned this pull request May 17, 2026

Add AzureBlobStorageDagBundle #66987

Open

yuseok89 reviewed May 17, 2026

View reviewed changes

Comment thread providers/microsoft/azure/src/airflow/providers/microsoft/azure/bundles/wasb.py Outdated

Comment thread providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/wasb.py Outdated

dominikhei reviewed May 18, 2026

View reviewed changes

Comment thread providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/wasb.py

Nishieee force-pushed the feature/azure-blob-dag-bundle branch from 0144712 to a16419b Compare May 19, 2026 17:28

Nishieee requested review from dominikhei, jroachgolf84 and yuseok89 May 20, 2026 15:33

jroachgolf84 suggested changes May 20, 2026

View reviewed changes

Comment thread providers/microsoft/azure/src/airflow/providers/microsoft/azure/bundles/wasb.py

Comment thread providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/wasb.py

Comment thread providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/wasb.py

Nishieee requested a review from jroachgolf84 May 24, 2026 01:46

jroachgolf84 approved these changes May 26, 2026

View reviewed changes

potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Jun 3, 2026

potiuk removed the ready for maintainer review Set after triaging when all criteria pass. label Jun 22, 2026

Nishieee and others added 7 commits June 25, 2026 10:31

add WasbDagBundle to load Dags from Azure Blob Storage

d8f7cd1

Use ValueError and NotADirectoryError in WasbDagBundle instead of Air…

edf61fd

…flowException

Docs: Add WasbDagBundle guide to Microsoft Azure provider

230b1a2

Update providers/microsoft/azure/src/airflow/providers/microsoft/azur…

2b5dacd

…e/bundles/wasb.py Co-authored-by: Yuseok Jo <yuseok89@gmail.com>

Add WasbHook.check_for_container tests and fix WasbDagBundle view URL…

50c1b92

… tests

Apply ruff-format to WasbHook tests

6baf8dc

Use ContainerClient.exists() in WasbHook.check_for_container

4070f0e

Add sync_to_local_dir tests for WasbHook

51d73e9

eladkal force-pushed the feature/azure-blob-dag-bundle branch from 2d3b4e3 to 51d73e9 Compare June 25, 2026 07:31

eladkal approved these changes Jun 25, 2026

View reviewed changes

shahar1 requested review from dabla and removed request for dominikhei and yuseok89 July 1, 2026 19:30

dabla requested changes Jul 2, 2026

View reviewed changes

Uh oh!

Conversation

Nishieee commented May 15, 2026

Summary

Was generative AI tooling used to co-author this PR?

Uh oh!

Uh oh!

jroachgolf84 commented May 16, 2026

Uh oh!

jroachgolf84 commented May 16, 2026

Uh oh!

Nishieee commented May 16, 2026

Uh oh!

yuseok89 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

potiuk commented May 18, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

potiuk commented May 26, 2026

Uh oh!

Nishieee commented May 26, 2026

Uh oh!

jroachgolf84 left a comment

Choose a reason for hiding this comment

Uh oh!

jroachgolf84 commented Jun 9, 2026

Uh oh!

eladkal left a comment

Choose a reason for hiding this comment

Uh oh!

eladkal commented Jun 25, 2026

Uh oh!

dabla Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

dabla Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

dabla Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

dabla Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

dabla Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants